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BACKGROUND OF THE INVENTION 

1. Field of the Invention 
[0001] 

The present invention relates to a technique for generating an image 
of high pixel density from a plurality of low pixel density images, and in 
particular to a technique for establishing an area for generating the image 
so that the resulting image is of higher quality. 

2. Description of the Related Art 
[ 0002 ] 

Conventional methods are available for synthesizing still an image 
of high pixel density from a plurality of frames of low pixel density motion 
images. Japanese Unexamined Patent Application (Kokai) 11-164264, for 
example, discloses a technique as follows. From a plurality of frame 
images for a device such as a CRT on which images are displayed by 
repeated scanning in the horizontal direction, a new image with a density 
greater than the density of the scanning lines of frame images in the 
vertical direction is generated. 
[ 0003 ] 

However, there are no techniques for determining an image 
generation area resulting in an image of higher quality when generating an 
image of high pixel density from a plurality of images having low pixel 
density. 
[ 0004 ] 

An object of the present invention, which was undertaken to address 
the above drawbacks in the prior art, is to determine an image generation 
area resulting in an image of higher quality when generating an image of 



high pixel density from a plurality of images having low pixel density. 
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SUMMARY OF THE INVENTION 

[ 0005 ] 

In order to address at least some of the above objects, the present 
invention employs the following process when generating an image of high 
5 pixel density from a plurality of images having low pixel density. First, a 
plurality of first images each of which includes a portion where a same 
recorded subject is recorded are prepared. An image generation area for 
generating a second image in which a density of pixels forming image is 
higher than that of the first images is determined, based on a overlap 
10 between the plurality of first images. Then the second image in the image 
generation area is generated from the plurality of first images. 
[ 0006 ] 

In the above aspect, the area which is included in many images 
among the plurality of the first images redundantly can be set as the image 
15 generation area. It is thus possible to determine an image generation area 
resulting in an image of higher quality when generating an image of high 
pixel density from a plurality of images having low pixel density. 
[ 0007 ] 

The following is preferred when determining the image generation 
20 area. The determination of the image generation area is executed so that 
an overlapping index value representing an extent of overlap between the 
plurality of first images and the image generation area is closest to a 
predetermined target level on a predetermined condition. In this aspect, 
the target level can be adjusted in order to determine an image generation 
25 area so that the evaluation of image generation area other than the extent 
of overlapping with the plurality of first images, e.g. the breadth of the 
image generation area, does not become poor. 
[ 0008 ] 

The following is preferred when determining the image generation 
30 area. That is, a plurality of candidate areas included in a sum area which 
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is sum of areas in which first images are recorded are first prepared. One 
of the candidate areas is selected as the image generation area from among 
the plurality of candidate areas, based on an evaluation value for each of the 
candidate areas which is determined based on overlaps between the 
5 plurality of first images and the candidate area. In this aspect, the image 
generation area can be selected from among limited candidates based on the 
evaluation value. An image generation area can thus be simply selected. 
[ 0009 ] 

When selecting the candidate area, it is preferable to determine the 
10 evaluation values for the candidate areas based on relative positions 
between the candidate areas and the first images. 
[0010] 

When selecting the candidate area, it is preferable to determine the 
evaluation value for each of the candidate areas. In the determination of 

15 the evaluation value for one of the candidate areas, the following is 
preferred. That is, an evaluation target portion is determined. The 
evaluation target portion is a portion of a profile of a target candidate area 
for which the evaluation value is being determined and included in an area 
of one of the plurality of first images. Then the evaluation value for the 

20 target candidate area is determined based on lengths of the evaluation 
target portions for the plurality of first images. In this aspect, an image 
generation area can be determined on the basis of simple calculations so as 
to result in an image of higher quality. 
[0011] 

25 When selecting the candidate area, the following embodiment may 

be employed. That is, sample points are set on a profile of each of the 
candidate areas. Then the evaluation values are determined for the 
candidate areas based on the sample points. In the determination of the 
evaluation value for one of candidate areas, the following is preferred. 

30 Evaluation sample points are determined among the sample points of a 
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target candidate area for which the evaluation value is being determined. 
The evaluation sample points are sample points included in an area of one 
of the plurality of first images. The evaluation sample points of the 
plurality of first images are determined. Then the evaluation value is 
5 determined for the target candidate area based on a number of the 
evaluation sample points of the plurality of first images. This aspect also 
allows an image generation area to be determined on the basis of simple 
calculations so as to result in an image of higher quality. 
[0012] 

10 When selecting the candidate area, the following embodiment may 

also be employed. Sample points are set on a profile of each of the first 
images. Then the evaluation values are determined for the candidate areas 
based on the sample points. In the determination of the evaluation value 
for one of candidate areas comprises, the following is preferable. That is, 

15 evaluation sample points are determined among the sample points of one of 
the first images. The evaluation sample points are sample points included 
in a target candidate area for which the evaluation value is being 
determined. Then the evaluation value is determined for the target 
candidate area based on numbers of the evaluation sample points of the 

20 plurality of first images. This aspect also allows candidate areas 
comprising an area of images including many overlapping first images to be 
selected as the image generation area based on simple calculations. 
[0013] 

When selecting the candidate area, following procedure may be 
25 executed. That is, evaluation areas having a certain width near profiles of 
the candidate areas are set. Then the evaluation values are determined for 
the candidate areas based on the evaluation areas. In the determination of 
the evaluation value for one of candidate areas the following is preferable. 
A limited evaluation area is determined. The limited evaluation area is a 
30 portion of a target candidate area for which the evaluation values is being 
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determined and is included in an area of one of the plurality of first images. 
Then a total number of pixels included in the limited evaluation area of the 
plurality of first images is calculated. The evaluation value is determined 
for the target candidate area based on total number of the pixels. 
5 [0014] 

When selecting the candidate area, following procedure may also be 
executed. That is, sample points are set near profiles of the candidate 
areas. Then the evaluation values for the candidate areas are determined 
based on the sample points. In the determination of the evaluation value 

10 for one of candidate areas, the following is preferable. Evaluation sample 
points are determined among the sample points of a target candidate area 
for which the evaluation value is being determined. The evaluation sample 
points are sample points included in an area of one of the plurality of first 
images. Then the evaluation value for the target candidate is determined 

15 based on a number of evaluation sample points for the plurality of first 
images. 
[0015] 

The following is also preferable. At least one of the plurality of first 
images is output through an output device. The second image is output 
20 through the output device in a same size as the first image output. In this 
aspect, user can compare the areas of the first and second images easily. 
[ 0016 ] 

In order to address at least some of the above objects, the following 
process can be employed when generating an image of high pixel density 

25 from a plurality of images having low pixel density. First, a plurality of the 
first images comprising portions of the same recorded subject, where the 
density of the pixels forming the images is relatively low, is prepared. The 
relative positions between the plurality of the first images are calculated 
based on the portions of the same recorded subject. An image generation 

30 area is then determined on the basis of the relative positions between the 
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plurality of first images. The image generation area is an area for 
generating a second image where the density of the pixels forming the 
image is relatively higher. The image generation area is to be included in a 
sum area comprising all the areas in which first images are recorded. In 
5 this aspect, the area of images comprising several overlapping first images 
among the plurality of first images can be set as the image generation area. 
An image generation area can thus be determined so as to result in an 
image of higher quality. 
[0017] 

10 In the determination based on the relative positions between the 

plurality of first images, the following is executed. First, a plurality of 
candidate areas included in the sum area comprising all the areas in which 
first images are recorded are first prepared. One of the candidate areas is 
then selected as the image generation area from among the plurality of 

15 candidate areas, based on an evaluation of each candidate area determined 
on the basis of the relative positions between the first images and the 
candidate areas. In this aspect, the image generation area can be simply 
selected based on the relative positions between the plurality of first images 
that have been prepared. 

20 [0018] 

When selecting the candidate area, it is preferable to determine the 
evaluation value based on numbers of pixels in the first images included in 
portions where the candidate area and the first images overlap. In this 
aspect, candidate areas including an area of images comprising many 
25 overlapping first images can be selected as the image generation area. An 
image generation area can thus be determined so as to result in an image of 
higher quality. 
[0019] 

When selecting candidate areas, evaluation values may be 
30 determined on the basis of the length of the portions in the first image areas 
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among the profile of the candidate areas. In this aspect, candidate areas 
comprising an area of images including many overlapping first images can 
be selected as the image generation area based on simpler calculations. 
That is, an image generation area can be determined based on simpler 
5 calculations so as to result in an image of higher quality. 
[ 0020 ] 

The evaluation values may also be determined based on the number 
of sample points included in the first image areas among the sample points 
set on the profile of the candidate areas when selecting candidate areas. In 

10 this aspect, candidate areas comprising an area of images including many 
overlapping first images can be selected as the image generation area based 
on even simpler calculations. That is, an image generation area can be 
determined on the basis of even simpler calculations so as to result in an 
image of higher quality. 

15 [0021] 

Evaluation values may also be determined on the basis of the 
number of sample points included in the candidate areas among the sample 
points set on the profile of the first images when selecting candidate areas. 
This aspect also allows candidate areas comprising an area of images 
20 including many overlapping first images to be selected as the image 
generation area based on simple calculations. 
[ 0022 ] 

When selecting candidate areas, the evaluation values may also be 
determined on the basis of the number of first image pixels included in 
25 portions in the first image areas among the evaluation areas near the profile 
of the candidate areas. 
[ 0023 ] 

Another aspect when selecting candidate areas is to determine the 
evaluation values based on the number of sample points included in the first 
30 image areas among the set sample points near the profile of the candidate 



areas. 
[ 0024 ] 

The following procedure is preferred when preparing the plurality of 
candidate areas. That is, a first candidate area included in the sum area 
5 being sum of areas in which first images are recorded is set first. Then a 
second candidate area and a third candidate area are prepared. The 
second candidate area is an area included in the sum area being sum of 
areas in which first images are recorded, and is to conform to the first 
candidate area by being displaced a certain extent in a first direction. The 

10 third candidate area is an area included in the sum area being sum of areas 
in which first images are recorded, and is to conform to the first candidate 
area by being displaced a certain extent in a direction opposite the first 
direction. In this aspect, the image generation area can be selected from 
among a plurality of candidate areas set in a certain range based on a first 

15 candidate area. 
[ 0025 ] 

The following procedure is preferred when preparing the plurality of 
candidate areas. First, a first candidate area included in the sum area 
being sum of areas in which first images are recorded is set. Then a second 

20 candidate area and a third candidate area are prepared. The second 
candidate area is an area included in the sum area being sum of areas in 
which first images are recorded, and is to conform to the first candidate area 
by being shrunk around a certain fixed point. The third candidate area is 
an area included in the sum area being sum of areas in which first images 

25 are recorded, and is to conform to the first candidate area by being 
magnified around a certain fixed point. This aspect allows the image 
generation area to be selected from prepared candidate areas that are larger 
or smaller than the first candidate area. The first candidate area is 
preferably indicated by the user. 

30 [ 0026 ] 
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The tone levels of the pixels in the second image may be calculated 
by the following procedure when generating a second image in cases where 
the pixels of the plurality of first images have varying tone levels. First, 
from pixels of the second image, a target pixel for calculating the tone level 
5 is selected. From the pixels of the plurality of first images, a plurality of 
specified pixels are selected. The specified pixels are pixels located in a 
certain range near the target pixel when the pixels of the plurality of first 
images are supposed to be arranged according to the relative positions and 
pixels of the second image are furthermore supposed to be arranged in the 
10 image generation area. Then tone level of the target pixel is calculated 
based on a weighted average of tone levels of the specified pixels. This 
aspect allows the tone levels of the pixels in an image of higher pixel density 
to be calculated from the tone levels of pixels in images of low pixel density. 
[ 0027 ] 

15 Specified pixels preferably include pixels closest to the target pixels 

among the plurality of first images when the pixels of the plurality of first 
images are arranged according to the relative positions and the second 
image pixels are furthermore arranged in the image generation area. 
Specified pixels may preferably be pixels included within a circle having a 

20 radius twice as long as the pitch of the first image pixels and the center 
identical with the target pixel when the pixels of the plurality of first images 
are arranged according to the relative positions and the second image pixels 
are furthermore arranged in the image generation area. 
[ 0028 ] 

25 The present invention can also be realized in the various aspects 

below. 

(1) Image-generating methods, image-processing methods, and 
image data* generating methods. 

(2) Image generators, image processors, image data generators. 

30 (3) Computer programs for running the above devices and methods. 
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(4) Recording media for recording computer programs for running 
the above devices and methods. 

(5) Data signals embodied in carrier waves and comprising computer 
programs for running the above devices and methods. 

5 [ 0029 ] 

These and other objects, features, aspects, and advantages of the 
present invention will become more apparent from the following detailed 
description of the preferred embodiments with the accompanying drawings. 

10 BRIEF DESCRIPTION OF THE DRAWINGS 

[ 0030 ] 

Figure 1 is a schematic illustration of the structure of an image 
generator in an embodiment of the invention; 

Figure 2 is a flow chart of the procedures for generating still image 
15 data representing a still image from a plurality of frame images of motion 
picture data; 

Figure 3 illustrates a user interface display screen for designating 
the instant in time for which a high resolution still image is desired by the 
user during motion picture playback; 
20 Figure 4 illustrates a method for specifying the relative positions in 

the frame image data; 

Figure 5 illustrates relative positions in five frames of image data Fl 
through F5; 

Figure 6 is a flow chart of a procedure for determining an area in 
25 which still image data is generated in Step S6 in Figure 2; 

Figure 7 illustrates candidate areas Acl to Acl2; 

Figure 8 illustrates the relative position between candidate areas 
AcO to Acl2 and an area Fa which is an area comprising all the areas in 
which frame image data Fl to F5 are recorded; 
30 Figure 9 illustrates the relationship between sample points Pe of 

candidate area AcO and the frame image data Fl through F5; 
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Figure 10 illustrates the number Nijk of sample points in the frame 
image data for candidate area AcO, the evaluation values S 0 j of each side of 
the candidate area AcO, and the evaluation value Eo of candidate area AcO; 

Figure 11 illustrates the relationship between image generation area 
5 Ad and portions in which the frame image data Fl through F5 overlap; 

Figure 12 illustrates a method for synthesizing an image of high 
pixel density from a plurality of images of low pixel density; 

Figure 13 is a flow chart of a procedure for determining the RGB 
tone levels for each pixel in an image of still image data based on the RGB 
10 tone levels for each pixel in the images of the frame image data; 

Figure 14 illustrates an evaluation area AeO which is set inside 
candidate area AcO and near the perimeter of candidate area AcO in a 
certain width; 

Figure 15 illustrates the length Lcoi of a portion contained in the 
15 area of the frame image data Fl within the four sides of candidate area AcO; 

Figure 16 illustrates sample points Pel through Pe5 set on the side 
of the image area of frame image data Fl through F5; 

Figure 17 is a schematic illustration of the structure of an image 
generator in Embodiment 5; 
20 Figure 18 is a flow chart of a procedure for generating still image 

data representing a still image from a plurality of frame images in motion 
picture data in Embodiment 5; and 

Figure 19 is an illustration of a user interface display screen 
displayed on a display 110 in Step S10 in Figure 18. 

25 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
[0031] 

The embodiments below illustrate the invention in the following 

order. 

30 A. Embodiment 1 
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A-l' Structure of Device 

A-2' Overall Procedure for Generating Still Image Data 
A-3- Determination of Image Generation Area 
A-4* Generation of Still Image Data 
5 B. Embodiment 2 

C. Embodiment 3 

D: Embodiment 4 

E: Embodiment 5 

F: Variants 

10 [ 0032 ] 

A. Embodiment 1 
A-l: Structure of Device 

Figure 1 is a schematic illustration of the structure of an image 
generator in an embodiment of the invention. The image generator 

15 comprises a personal computer 100 for running certain image processing on 
the image data, a keyboard 120, a mouse 130 and CD-R/RW drive 140 as 
devices for inputting data to the personal computer 100, and a display 110 
and printer 22 as devices for outputting data. An application program 95 is 
operated under the control of a certain operating system in the computer 

20 100. The application program 95 is run to allow the CPU 102 of the 
computer 100 to execute various functions. 
[ 0033 ] 

When an application program 95 for retouching images or the like is 
run and the user inputs commands via the keyboard 120 or mouse 130, the 
25 CPU 102 reads image data into memory from the CD-RW in the CD-R/RW 
drive 140. The CPU 102 runs a certain image process on the image data 
and displays an image through a video driver on the display 110. The CPU 
102 can print the processed image data via a printer driver to a printer 22. 
[ 0034 ] 

30 Image data comprising motion pictures includes a plurality of frame 
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image data, each of which represents a still image. The plurality of frame 
image data is consecutively numbered, and the still image of each frame 
image data is displayed on the display 110 according to the consecutive 
sequence to playback the motion pictures on the display 110. 
5 [ 0035 ] 

A-2: Overall Procedure for Generating Still Image Data 
Figure 2 is a flow chart of the procedure for generating still image 
data representing the still images from the plurality of frame images in the 
motion picture data. In Step S2, when the application program 95 is run 
10 and the user inputs commands by way of the keyboard 120 or mouse 130, 
the CPU 102 first obtains five continuous frames of image data from the 
image data representing the motion pictures stored in memory. 
[ 0036 ] 

Figure 3 illustrates a user interface display screen for designating 
15 the instant in time for which a high resolution still image is desired by the 
user during motion picture playback. For example, the CPU 102 reads 
certain motion picture data from the CD-RW (movie file Movie.avi in the 
example in Figure 3) and stores it in memory based on user commands 
input by the keyboard 120 or mouse 130 to the computer 100. As 
20 illustrated in Figure 3, the motion picture Fm of the image data is played 
back on the display 110. In Step S2, when the motion picture is played 
back on the display 110, the user moves the cursor Cs via the mouse 130 and 
presses the "scene capture" button in the user interface display screen to 
designate a specific instant during motion picture playback. The keyboard 
25 120 can also be used to designate the specific instant during motion picture 
playback. 
[ 0037 ] 

When the user designates the specific instant during motion picture 
playback, the CPU 102 obtains the frame image data F3 displayed on the 
30 display 110 at that instant, the previous two frames of motion picture data 



14 



Fl and F2, and the next two frames of image data F4 and F5. In this way, 
the function of obtaining a plurality of frames of image data as instructed by 
the user is executed by a frame data capturing component 102a (see 
Figure l) which is a functional component of the CPU 102. 
5 [ 0038 ] 

Let us assume that the motion picture data read from the CD-RW 
and stored in memory is motion picture data with a 3-4 aspect ratio, and 
that the motion picture data is of a still object, such as a landscape or still 
life, which is slightly swayed by the hand movements of the individual 
10 taking the picture. The subject will therefore be the same in the still 
pictures represented by each of the five frames of image data selected in 
Step S2, but the position of the photographed subject in the images will be 
slightly displaced. 
[ 0039 ] 

15 Figure 4 illustrates a method for specifying the relative positions in 

the frame image data. In Step S4 in Figure 2, the displacement of the 
relative positions between the images in the five frames of image data read 
in Step S2 is calculated. The relative positional displacement of the images 
in the frames of image data is determined in the following manner. 

20 [ 0040 ] 

First, characteristic points are determined in the portions where the 
same image is recorded in the images. In Figure 4, the characteristic 
points are represented by the black solid circles Spl through Sp3 in the 
frames of image data Fl and F3. In Figure 4, two mountains and the sky 

25 are the same subject in the frames of image data Fl and F3. The 
characteristic points are located at characteristic image sections which do 
not frequently appear in common images. As illustrated in Figure 4, for 
example, the mountain peaks (Spl and Sp3) or the point where the profiles 
of the mountains intersect (Sp2) can be used. 

30 [ 0041 ] 
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Then, as illustrated in the bottom drawing in Figure 4, the relative 
positional displacement between the images in the frames of image data Fl 
and F3 is specified and calculated by determining the relative positions 
between the images in the frames of image data so that the characteristic 
5 points Spl through Sp3 in the frames of image data Fl and F3 overlap. 
[ 0042 ] 

Figure 5 illustrates relative positions in five frames of image data Fl 
through F5. When the relative positional displacement between the five 
frames of image data Fl through F5 is calculated in Step S4, the relative 

10 positions between the frames of image data Fl through F5 are specified as 
illustrated in Figure 5. In the image illustrated in Figure 5, p5 represents 
the portion where the same images is recorded when all five frames of image 
data Fl through F5 over lap. The symbols p2 through p4 indicate where 
the same image is recorded when two to four frames of image data overlap. 

15 The symbol pi indicates where only one frame of image data is recorded. 
[ 0043 ] 

The function of specifying the relative positions between the images 
in the plurality of frames of image data based on the characteristic points is 
managed by a frame synthesizer 102b, which is a functional component of 

20 the CPU 102. The displacement of the relative positions between the 
frames of image data Fl through F5 in Figure 5 has been exaggerated for 
the convenience of explanation. That is, Figure 5 does not reflect the 
actual extent of displacement between the images in the motion picture 
frames. 

25 [ 0044 ] 

In Step S6 of Figure 2, it is determined that which portion in the 
images represented by the frames of motion picture data Fl through F5 will 
be used to generate a still image. The image to be generated, represented 
by still image data, has the same rectangular aspect ratio of 3^4 as the 
30 images of the motion picture data. The pixel density of the image of the 
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still image data to be generated is four times greater than the vertical and 
horizontal pixel density of the images in the frames of motion picture data 
Fl through F5. The image area where the still image data is generated is 
referred to as the "image generation area." The function of determining the 
5 image generation area where the still image data is generated is managed 
by an image generation determining component 102c (see Figure l) which is 
a functional component of the CPU 102. 
[ 0045 ] 

Then, in Step S8, the still image data is generated for the area 
10 determined in Step S6. The function of generating the still image data is 
managed by a still image generator 102d (see Figure l) which is a functional 
component of the CPU 102. The procedure for determining the image 
generation area in Step S6 and generating the still image data in Step S8 is 
described below. 
15 [ 0046 ] 

A-3- Determination of Image Generation Area 

Figure 6 is a flow chart of a procedure for determining an area in 
which still image data is generated in Step S6 in Figure 2. In Step S22, a 
target evaluation value St is first set. St is any number from 1 to 5. The 

20 target evaluation value St can be any value from 1 to the "number of frames 
of image data obtained in Step S2." The greater the target evaluation value 
St, the greater the possibility of generating a still image with greater 
resolution but also the greater possibility of a smaller image generation area. 
The target evaluation value St is described below. 

25 [ 0047 ] 

The target evaluation value St may be a pre-determined value such 
as 4 or 3, or the user may input a level to the computer 100 through the 
mouse 130 or keyboard 120. When the user sets the target evaluation 
value St, the user can control the balance between the resolution and the 
30 size of the image generation area of the still image that is produced by 
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adjusting the target evaluation value St. 
[ 0048 ] 

In Step S24, candidate areas AcO through Acl2 which are candidates 
for the image generation area are set. The function of generating the 
5 candidate area is managed by a candidate area generator 102e (see Figure 
l) which is a functional component of the CPU 102. The candidate area 
generator 102e is a functional component constituting a part of the 
generation area determining component 102c which is a functional 
component of the CPU 102. 
10 [ 0049 ] 

In Step S24, candidate area AcO is first set. The candidate area AcO 
is equivalent to the area of the image in frame image data F3 (see Figure 5). 
The aspect ratio of the candidate area AcO is 3-4. 
[ 0050 ] 

15 Figure 7 illustrates candidate areas Acl to Acl2. The dashed lines 

in the figure represent the relative position of the candidate area AcO in 
relation to candidate areas Acl to Acl2. Candidate area Acl is an area 
displaced 1 pixel upward relative to candidate area AcO which is equivalent 
to the area of frame image data F3. 

20 [ 0051 ] 

The "1 pixel" referred to here is 1 pixel in the pixel density of the 
frame image data, and is not 1 pixel in the pixel density of the still image 
data to be generated (4 times the pixel density of the frame image data). 
Thus, stated in terms of the units of pixels for the pixel density in the still 

25 image data, candidate area Acl is an area displaced upward 4 pixels relative 
to candidate area AcO. The extent to which the candidate areas are 
displaced is illustrated disproportionately to the actual dimensions in 
Figure 7 in order to facilitate the explanation of the relative positions 
between candidate area AcO and candidate areas Acl through Acl2. 

30 [ 0052 ] 
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Candidate area Ac2 is an area displaced 1 pixel down relative to 
candidate area AcO. Candidate area Ac3 is an area displaced 1 pixel left 
relative to candidate area AcO, and candidate area Ac4 is an area displaced 
1 pixel right relative to candidate area AcO. That is, candidate area Ac3 
5 can be displaced 1 pixel to the right relative to candidate area AcO to overlap 
candidate area AcO. Candidate area Ac2 can be displaced 1 pixel to the left 
relative to candidate area AcO to overlap candidate area AcO. The hollow 
arrows in the figure indicate the directions in which candidate areas Acl 
through 4 are displaced relative to candidate area AcO. 
10 [ 0053 ] 

The area of candidate area Ac5 is 1 pixel short at the left end 
relative to candidate area AcO and % pixel short at the bottom end. The 
aspect ratio of candidate area Ac5 is thus 3^4, the same as that of candidate 
area AcO. That is, candidate area Ac5 is an area in which candidate area 
15 AcO is shrunk, where the apex at the upper right is the reference point. 
[ 0054 ] 

The "1 pixel" referred to here is 1 pixel in the pixel density of the 
frame image data, and is not 1 pixel in the pixel density of the still image 
data to be generated. Thus, stated in terms of the units of pixels for the 
20 pixel density in the still image data, candidate area Ac5 is an area lacking 4 
pixels at the left end relative to candidate area AcO and lacking 3 pixels at 
the bottom end. 
[ 0055 ] 

Candidate area Ac6 is an area short 1 pixel at the right end relative 
25 to candidate area AcO and is short % pixel at the bottom end. Candidate 
area Ac7 is an area short 1 pixel at the right end relative to candidate area 
AcO and is short % pixel at the top end. Candidate area Ac8 is an area 
short 1 pixel at the left end relative to candidate area AcO and is short % 
pixel at the top end. The aspect ratio of candidate areas Ac6 to 8 are 3-4 in 
30 the same manner as in candidate area AcO. In the figure, the arrows in 
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candidate areas Ac5 to 8 indicate the directions in which candidate areas 
Ac5 to 8 are shrunk relative to candidate area AcO. 
[ 0056 ] 

Candidate area Ac9 is an area expanded 1 pixel at the right end 
5 relative to candidate area AcO and expanded % pixel at the top end. 
Candidate area AclO is an area expanded 1 pixel at the left end relative to 
candidate area AcO and expanded % pixel at the top end. Candidate area 
Ac 11 is an area expanded 1 pixel at the left end relative to candidate area 
AcO and expanded % pixel at the bottom end. Candidate area Acl2 is an 
10 area expanded 1 pixel at the right end relative to candidate area AcO and 
expanded % pixel at the bottom end. 
[ 0057 ] 

The aspect ratio of these candidate areas Ac9 through 12 is 3-4 in the 
same way as in candidate area AcO. The arrows near the outer periphery of 

15 candidate areas Ac9 through Acl2 indicate the directions in which the 
candidate areas Ac9 through Acl2 are expanded relative to candidate area 
AcO. The angle diagonal to the angle indicated by these arrows is the 
reference point in the expansion or shrinkage of the candidate areas relative 
to candidate area AcO. 

20 [ 0058 ] 

Candidate areas Ac5 to Acl2 comprising the expansion or shrinkage 
of candidate area AcO as described above all have a rectangular aspect ratio 
of 3*4. It is thus possible to generate an image with the same aspect ratio 
as the motion pictures no matter which of the candidate areas is selected as 

25 the image generation area. The still image data generated in Step S8 in 
Figure 2 is composed with a pixel density 4 times greater than that of frame 
image data Fl through F5. The image that is generated can thus be 
expressed as the aggregation of pixels of still image data through the 
expansion or shrinkage of the vertical dimensions in units equal to V4 the 

30 pixels of the frames of image data, as in candidate areas Ac5 through Acl2. 
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[ 0059 ] 

Figure 8 illustrates the relative positions between candidate areas 
AcO to Acl2 and an area Fa which is an area comprising all the areas in 
which frame image data Fl to F5 are recorded. The area Fa in Figure 8 is 
5 the convergence of areas included in any of the areas in the images of the 
frames of image data Fl through F5. In Figure 8, candidate area AcO is 
indicated by solid lines, while candidate areas Acl through Acl2 are 
indicated by dashed lines. The hollow arrows indicate the directions in 
which candidate areas Acl through Acl2 move, expand, or shrink in relation 

10 to candidate area AcO. In the first example, candidate areas Acl through 
Ac4 (which were areas shifted up, down, left, or right, as shown in Figure 8, 
based on candidate area AcO (the area selected by the user in Step S2 of 
Figure)) and candidate areas Ac5 to Acl2 (which were expanded or shrunk 
areas) were set as the plurality of candidate areas. One candidate area is 

15 selected as the image generation area Ad from among these candidate areas 
Ac lthrough Acl2. An image close to the image desired by the user can 
thus be generated in the form of a still image. 
[ 0060 ] 

For example, candidate area Acl is an area displaced 1 pixel upward 
20 relative to candidate area AcO, and candidate are Ac2 is an area displaced 1 
pixel down relative to candidate area AcO. That is, candidate area Acl can 
be shifted 1 pixel down relative to candidate area AcO to overlap candidate 
area AcO, and candidate area Ac2 can be shifted 1 pixel up relative to 
candidate area AcO to overlap candidate area AcO. It is thus possible to 
25 select a desirable image generation area while respecting the image area 
designated by the user by preparing candidate areas in which the candidate 
area AcO (the image area indicated by the user) is shifted in mutually 
opposed directions. 
[ 0061 ] 

30 Also, for example, candidate area Ac5 is an area in which candidate 
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area AcO is shrunk using the apex on the upper right as the reference point. 
By contrast, candidate area Acllis an area in which candidate area AcO is 
expanded using the apex on the upper right as the reference point. In 
other words, candidate area Ac5 can be expanded using the apex at the top 
5 right as a reference point to overlap candidate area AcO. Candidate area 
Acll can be shrunk using the apex at the top right as a reference point to 
overlap candidate area AcO. It is thus possible to select a desirable image 
generation area while respecting the image area designated by the user by 
preparing candidate areas in which the candidate area AcO (the image area 
10 indicated by the user) is expanded or shrunk using the same reference point. 
[ 0062 ] 

In Embodiment 1, the same number of candidate areas shifted in 
opposed directions based on candidate area AcO (one each in Embodiment l) 
were used as candidate areas. A still image can thus be generated with an 

15 area in which an image of high pixel density is readily generated, being an 
area close to the image area desired by the user. Similarly, the same 
number of candidate areas comprising areas expanded or shrunk based on 
the same reference point with respect to candidate area AcO (one each in 
Embodiment l) were set as candidate areas. A still image can thus be 

20 generated with an area in which an image of high pixel density is readily 
generated, being an area close to the image area desired by the user. 
[ 0063 ] 

In Step S26 in Figure 6, a candidate area for which the evaluation 
value El is to be estimated is selected from candidate areas AcO to Acl2. 
25 First, candidate area AcO equivalent to the image area of frame image data 
F3 is selected. Frame image data F3 is the frame image data with the 
instant which the user specified by keyboard 120 or mouse 130 during 
motion picture playback in Step S2 of Figure 2. 
[ 0064 ] 



22 



Figure 9 illustrates the relationship between sample points Pe of 
candidate area AcO and the frame image data Fl through F5. Five sample 
points Pc are set on each side of candidate areas AcO to Acl2. Each sample 
point is set equidistantly within the sides, and the distance from the sample 
5 points at both ends to the sides is V2 the distance between the sample points. 
[ 0065 ] 

In Step S28 of Figure 6, the number of sample points Nijk which are 
within each frame image data out of all sample points on the sides of the 
candidate area selected in Step S26. Here, i is an integer from 0 to 12 

10 corresponding to candidate area AcO to Acl2, and j is an integer from 1 to 4 
representing the four sides of the rectangular candidate areas, where j = 1 
means the left side, j = 2 means the right side, j= 3 means the top side, and j 
= 4 means the bottom side. The symbol k is an integer from 1 to 5 
corresponding to frame image data Fl through F5. In the present 

15 Specification, sample points which are within the area of the frame image 
data out of sample points set in relation to the candidate areas are referred 
to as "evaluation sample points." When the process first reaches Step S28, 
the number Nijk for the objects in the frame image data among the sample 
points on the left side of the candidate areas is first determined. 

20 [ 0066 ] 

Figure 10 illustrates the number Nijk of sample points in^the frame 
image data for candidate area AcO, the evaluation values S 0 j of the candidate 
area AcO, and the evaluation value Eo of candidate area AcO. For example, 
in Figure 9, four of the five sample points Pe on the left side of candidate 

25 area AcO are in the image area of frame image data Fl (indicated by the 
relatively rough dashed line). Thus, 4 is indicated in the "Fl" column in 
the "left side" row in Figure 10. Similarly, five of the sample points Pe on 
the left side of candidate area AcO are in frame image data F2 (indicated by 
relatively narrow dashed line). Thus, 5 is indicated in the "F2" column in 

30 the "left side" row in Figure 10. 
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[ 0067 ] 

Because candidate area AcO is consistent with the frame image data 
F3, the sample points Pe on each side of candidate area AcO are on each side 
of the frame image data F3. When the sample points of a candidate area 
5 are on each side of frame image data, those sample points are regarded as 
"not" being in the frame image data. Thus, 0 is indicated in column "F3" in 
the "left side" row in Figure 10. Because none of the sample points on the 
sides of the candidate area AcO are regarded as being in the frame image 
data F3, sample point numbers N013, N023, N033, and N043 are also 0 in the 
10 "right," "upper," and "bottom side" rows as well as in the "left side" row of 
column "F3" in Figure 10. 
[ 0068 ] 

As noted above, in Step S28 of Figure 6, the number Nijk of sample 
points on the sides of the candidate area selected in Step S26 which are also 

15 in frame image data is determined. That is, when the candidate area AcO 
is selected in Step S26, the values for all of the left, right, upper, and bottom 
side rows in each column Fl through F5 in Figure 10 are determined for the 
candidate area AcO. The values for the left side row of columns Fl through 
F5 in Figure 10 are determined first. 

20 [ 0069 ] 

In Step S30 of Figure 6, the evaluation values Sij for the sides of the 
candidate areas are first determined. The evaluation values Sij for the 
sides are calculated by the following Equation (l). NijA is the total number 
of sample points set for the sides of the candidate areas. In Embodiment 1, 
25 NijA is 5 for all sides. 
[ 0070 ] 




[0071] 
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With regard to the left side of candidate area AcO, for example, as 
shown in Figure 10, there are 4 sample points in frame image data Fl, 5 
sample points in frame image data F2, and 0 sample points in frame image 
data F3 through F5, so Soi is 1.8. The other evaluation values SOI to S04 
5 for the sides of candidate area AcO are given in the table in Figure 10. 
[ 0072 ] 

In Step S32 of Figure 6, it is determined whether evaluation values 
Sij have been calculated for all the sides of the candidate area selected in 
Step S26. When the result is No because evaluation values Sij have not 

10 been calculated for all the sides of the candidate area, the process returns to 
Step S28. The number Nijk for points in the frame image data is 
determined for each side for which no evaluation value Sij has been 
calculated. Steps S28 through S32 are repeated until evaluation values Sij 
have been calculated for all the sides of the candidate area selected in Step 

15 S26. The process proceeds to Step S34 once the results in Step S32 are 
determined to be Yes. 
[ 0073 ] 

In Step S34, the evaluation value Ei for the candidate area Aci 
selected in Step S26 (i is the number designating the candidate area* i = 0 to 
20 12) is determined. The evaluation value Ei is calculated by the following 
Equation (2). St is the target evaluation value set in Step S22. 
[ 0074 ] 

Ei = Z(Sy-St) 2 • • • (2) 

[ 0075 ] 

25 Eo is 17.68 in the example shown in Figures 9 and 10. As will be 

evident from the form of Equation (2), the closer the evaluation values Sij of 
each side are to the target evaluation value St set in Step S22, the lower the 
evaluation value Ei of the candidate area Aci. As will be evident from 
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Equations (l) and (2), the evaluation value Ei is determined on the basis of 
the target evaluation value St and the number Nijk for the sample points in 
the frame image data (evaluation sample points). Here, the number of 
evaluation sample points is determined by the overlap between the 
5 candidate area and the first images. The evaluation value Ei is thus 
determined on the basis of the target evaluation value St and the overlap 
between candidate areas and the first images. 
[ 0076 ] 

In Step S36, it is determined whether the evaluation value Ei has 
10 been determined for all candidate areas AcO to Acl2. When the result is No 
because there are some candidate areas for which the evaluation value Ei 
has not been calculated, the process returns to Step S26, and the next 
candidate area is set from among the candidate areas for which no 
evaluation value Ei has been calculated. The process proceeds to Step S38 
15 when the evaluation value Ei is calculated for all candidate areas AcO to 
Acl2. 
[ 0077 ] 

In Step S38, the candidate area with the lowest evaluation value Ei 
is selected as the image generation area. That is, the candidate area in 
20 which the evaluation value Sij for each side is near the target evaluation 
value St most is selected as the image generation area. The process for 
determining the image generation area (Step S6 in Figure 2) is thus 
concluded. 
[ 0078 ] 

25 The function of calculating the evaluation values of the candidate 

areas and selecting one candidate area from among the plurality of 
candidate areas based on this evaluation value is managed by a candidate 
area selector 102f (see Figure l) which is a functional component of the CPU 
102. The candidate area selector 102f is a functional component 

30 constituting part of the generation area determining component 102c which 
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is a functional component of the CPU 102. 
[ 0079 ] 

Figure 11 illustrates the relationship between image generation area 
Ad and portions in which the frame image data Fl through F5 overlap. In 
5 Figure 11, the portion p5 which is recorded in the frame image data Fl 
through F5 redundantly is indicated by fine vertical-horizontal 
cross-hatching, and the portion p4 which is recorded in four overlapping sets 
of frame image data is indicated by diagonal cross-hatching. The portion 
p3 which is recorded in three overlapping sets of frame image data is 
10 indicated by slanted hatching, and the portion p2 which is recorded in two 
sets of overlapping frame image data is represented by coarse 
cross-hatching. 
[ 0080 ] 

The selection of the image generation area from among the 
15 candidate areas in the manner described above enables that a candidate 
area including the most sample points in the fame image data Fl through 
F5 is selected as the image generation area Ad. 
[ 0081 ] 

The candidate area including the most sample points in the frame 
20 image data Fl through F5 includes the area from the areas of frame image 
data most. When such a candidate area is used as the image generation 
area, the tone levels of the pixels in the still image can be properly specified 
based on pixel values of many pixels in many frame image data when 
generating the still image described below. 
25 [ 0082 ] 

In the example in Figure 11, for example, candidate area Ac5 in 
which the lower end and left end are shrunk relative to candidate area AcO 
(equivalent to the area in frame image data F3) is selected as the image 
generation area Ad. In Figure 11, the area of portions p5 and p4 are both 
30 contained in the image generation area Ad. The lower left part of portion 
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pi is contained in the image generation area Ad, but the rest of portion pi is 
not contained in the image generation area. The lower left and upper right 
parts of portion p2 are contained in the image generation area Ad, but the 
rest of portion p2 is not contained in the image generation area Ad. 
5 [ 0083 ] 

Candidate area Ac5 is assumed to be selected as the image 
generation area Ad for the convenience of explanation. This assumption 
therefore does not mean that candidate area Ac5 is selected according to the 
procedure in the flow charts based on the relationship between the sample 
10 points Pe and frame image data Fl through F5 in Figures 5 or 9. 
[ 0084 ] 

In Embodiment 1, evaluation values El were calculated for a limited 
number of candidate areas, and the image generation area was selected 
from the candidate areas based on those grades. It is therefore possible to 
15 determine an image generation area capable of properly specifying the tone 
levels of the pixels in the still image in a short time. 
[ 0085 ] 

In Embodiment 1, candidate areas with areas displaced up, down, to 
the left, and to the right based on the candidate area AcO selected by the 
20 user in Step S2 of Figure 2, and candidate areas with expanded or shrunk 
areas, were set as candidate areas. It is thus possible to generate an image 
close to the one desired by the user in the form of a still image. 
[ 0086 ] 

A-4- Generation of Still Image Data 

25 In Step S8 in Figure 2, still image data is generated based on the 

frame image data Fl through F5 for the area determined in Step S6. The 
frame image data are data representing the image as a aggregation of pixels. 
The pixels have red (R), green (G), and blue (B) tone levels. That is, the 
pixels comprise data on RGB tone levels and data on their positions in the 

30 image of each set of frame image data. 
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[ 0087 ] 

Figure 12 illustrates a method for synthesizing an image of high 
pixel density from a plurality of images of low pixel density. The circled 
numeral l's in the figure indicate the central positions of pixels in frame 
5 image data Fl. The circled numeral 2's indicate the central positions of 
pixels in frame image data F2. The plus signs indicate the central 
positions of pixels in the still image data to be generated. To simplify the 
description, the frame image data is limited to Fl and F2. Only some of 
the pixels of frame image data Fl and F2 and of the still image data are 
10 shown in Figure 12. The central positions of the pixels doe not reflect the 
actual relative positions of the image generation area determined in Step S6 
or the area of pixels in the frame image data Fl and F2 shown in Figure 5. 
[ 0088 ] 

As noted above, the pixel density of the still image data is four times 
15 that of the frame image data. The intervals between the plus signs in 
Figure 12 are thus X A the distance between the circled l's and between the 
circled 2's. 
[ 0089 ] 

Figure 13 is a flow chart of a procedure for determining the RGB 
20 tone levels for each pixel in an image of still image data based on the RGB 
tone levels for each pixel in the images of the frame image data. To 
generate still image data from the frame image data, the RGB tone levels 
must be determined for the positions shown by the plus signs in Figure 12. 
This is done by the following procedure. 
25 [ 0090 ] 

First, in Step S52, a target pixel for calculating the tone levels is 
specified. The target pixel for calculating the tone level in this case is Psl 
in Figure 12. In Step S54, a pixel of which a central position is closest to 
the target pixel Psl among the pixels in frame image data Fl and F2 is 
30 specified. This pixel is referred to as the "nearest pixel." In Figure 12, the 
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nearest pixel is Pnll indicated by the double circled 2 among the pixels of 
frame image data F2. 
[ 0091 ] 

After the nearest pixel Pnll is specified, three neighboring pixels of 
5 the nearest pixel Pnll are specified, which are pixels in the frame image 
data including the nearest pixel Pnll and surround the target pixel Psl 
with nearest pixel Pnll. In this example, these pixels, including the 
nearest pixel, are referred to as the "specified pixels." In the example in 
Figure 12, the four pixels Pnll, Pnl2, Pnl3, and Pnl4 at the top left among 
10 the pixels in frame image data F2 are the specified pixels. 
[ 0092 ] 

Then, in Step S58, the tone level of the subject Psl is calculated 
based on the weighted average. Specifically, the tone level Vt of the target 
pixel Psl can be determined by the following Equation (3), where VI to V4 

15 are red, green, or blue tone levels of the specified pixels Pnll through Pnl4, 
respectively, and rl through r4 are constants. Vt can be determined by 
Equation (3) from the red tone levels VI, V2, V3, and V4 of the specified 
pixels, where Vt is, for example, the red tone level of the target pixel Psl. 
The tone levels of the target pixel are calculated for red, green, and blue. 

20 [ 0093 ] 

Vt = (rl X vl) + (r2 X v2) + (r3 :>< v3) + (r4 X v 4) ... (3) 

[ 0094 ] 

Here, rl through r4 can be determined by Equations (4) through (7) 
below. Aa is the surface area of the rectangle surrounded by the four 

25 specified pixels Pnll through Pnl4. Al is the area of the quadrangle 
composed of the target pixel Psl and the three specified pixels Pnl2 through 
Pnl4 other than Pn 11. Similarly, A2 is area of the quadrangle composed of 
the target pixel Psl and the three specified pixels other than Pnl2. A3 is 
area of the quadrangle composed of the target pixel Psl and the three 

30 specified pixels other than Pnl3. A4 is area of the quadrangle composed of 
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the target pixel Psl and the three specified pixels other than Pnl4. 
[ 0095 ] 

rl=Al/Aa ... (4) 

[ 0096 ] 

5 r2=A2/Aa ... (5) 

[ 0097 ] 

r3=A3/Aa ... (6) 

[ 0098 ] 

r4=A4/Aa ... (7) 

10 [ 0099 ] 

In Step S60 of Figure 13, it is determined whether tone levels have 
been calculated for all the pixels of the still image data. The process 
returns to Step S52 when the result is No because there are pixels for which 
the tone level has not been calculated. 
15 [0100] 

When the tone level is calculated for pixel Ps2 in Figure 12, for 
example, the nearest pixel is Pn21 indicated by the double circled 1. The 
specified pixels are Pn21 through Pn24 in frame image data Fl. The tone 
levels of pixel Ps2 can be calculated from the tone levels of the specified 
20 pixels Pn21 through Pn24 based on Equation (3) in the same manner as 
pixel Psl. 
[0101] 

In Step S60 of Figure 13, the process is ended when the result is Yes 
because tone levels have been calculated for all pixels. 
25 [0102] 

The above procedure can be carried out to generate still image data 
of relatively high pixel density from a plurality of frame image data of 
relatively low pixel density. Tone levels can be determined at values close 
to the actual color because the tone levels of the nearest pixel close to the 
30 target pixel for which the tone levels are calculated are most reflected, and 
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the pixel values of other pixels near the nearest pixel are used for 

interpolation. 

[0103] 

B. Embodiment 2 

5 In Embodiment 1, as illustrated in Figure 9, sample points Pe on the 

sides of a candidate area were set, and an evaluation value Ei for the 
candidate area was determined based on the number of sample points Pe 
included in frame image data Fl through F5. The image generation area 
was then determined from among the plurality of candidate area based on 
10 the evaluation value Ei. In Embodiment 2, the method for selecting a 
candidate area as the image generation area from the plurality of candidate 
areas is different than in Embodiment 1. The other points are the same as 
in Embodiment 1. 
[0104] 

15 Figure 14 illustrates an evaluation area AeO which is set inside 

candidate area AcO and near the perimeter of candidate area AcO in a 
certain width. In Embodiment 2, an evaluation area Aei (i is the number 
designating the candidate area- i = 0 to 12) is set at a certain width near the 
outer periphery in a candidate area Aci. Here, the width of the evaluation 

20 area Aei is 1/20 of the long side of the candidate area. In Figure 14, the 
evaluation area AeO is the area indicated by the hatching and 
cross-hatching. 
[0105] 

When determining the evaluation value Di of the candidate area Aci 
25 (i is the number designating the candidate area- i = 0 to 12), the number of 
pixels Tii (i is the number designating the candidate area' i = 0 to 12) in the 
area (represented by the cross-hatching in Figure 14) included in the frame 
image data Fl within the evaluation area Aei is calculated. Here, the 
pixels counted for the evaluation area Aei are pixels in the frame image data 
30 Fl. 
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[ 0106 ] 

Similarly, the number of pixels Ti2 to Tis in frame image data F2 
through F5 in the portion included in frame image data F2 through F5 
within the evaluation area Aei are calculated. The evaluation value Di of 
5 the candidate area is determined by Equation (8) below. Here, Ta is the 
number of pixels in the frame image data included in the evaluation area 
when the evaluation area is consistent with the area of the frame image 
data. Constants i and k are the same as in Embodiment 1. In the present 
Specification, the portion of the evaluation area Aei contained in the frame 
10 image data area is referred to as the "limited evaluation area." 
[0107] 

[ 0108 ] 

The candidate area with the greatest Di is selected as the image 
15 generation area. This embodiment also allows candidate areas including 
many areas with several overlapping sets of frame image data to be selected 
as the image generation area. That is, in this embodiment, the image 
generation area is an area capable of generating high resolution still 
images. 
20 [ 0109 ] 

C. Embodiment 3 

In Embodiment 2, the evaluation value Di of the candidate areas Aci 
was determined based on the number of pixels in the area contained in the 
frame image data Fl within the evaluation area Aei. An image generation 
25 area was then determined from among the candidate areas Aci based on the 
evaluation value Di. In Embodiment 3, the image generation area is 
determined from among the candidate areas Aci based on the length Lcik of 
sections contained in the frame image data within the sides of the candidate 
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areas Aci. The other points are the same as Embodiment 2. In the 
present Specification, the portion included in the frame image data area 
within the sides of the candidate areas Aci is referred to as the evaluation 
target portion. 
5 [0110] 

Figure 15 illustrates the length Lcoi of a portion contained in the 
area of the frame image data Fl within the four sides of candidate area AcO. 
In Embodiment 3, the evaluation values Gi (i is the number designating the 
candidate area' i = 0 to 12) of the candidate areas Aci are determined by 

10 Equation (9) below. Lcik is the length of the portion contained in the area of 
frame image data within the 4 sides around candidate areas Aci. The 
constants i and k are the same as in Embodiment 1. Lcik corresponds to 
the "evaluation distance" in the "Means for Solving the Abovementioned 
Problems." LI is the length of the short side of the candidate areas, and L2 

15 is the length of the long side. 
[0111] 



[0112 I 

The candidate area with the greatest Gi is selected as the image 
20 generation area. This embodiment allows a candidate area with more 
areas of several overlapping frame image data to be selected as the image 
generation area. That is, in this embodiment, an area capable of 
generating a high resolution still image can be used as the image generation 
area. 
25 [0113] 

D. Embodiment 4 

In Embodiment 4, the method for selecting a candidate area as the 
image generation area from a plurality of candidate areas is different than 
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in Embodiment 1. The other points are the same as Embodiment 1. 
[0114] 

Figure 16 illustrates sample points Pel through Pe5 set on the side 
of the image area of frame image data Fl through F5. The sample points 
5 are set equidistantly on the sides around areas in the frame image data Fl 
through F5. The distance from the sample points at either end within a 
side to the side are % the distance between the sample points. 
[0115] 

In Embodiment 4, the evaluation value Hi (i is the number 
10 designating the candidate area- i = 0 to 12) of the candidate areas Aci is the 
number of sample points Pel through Pe5 which the candidate areas 
include. In Figure 16, the sample points Pel through Pe5 contained in the 
candidate area AcO are designated by rings around black circles. In the 
present Specification, the sample points on the sides of the frame image 
15 data which are included in the candidate areas are referred to as 
"evaluation sample points." In the example in Figure 16, 57 is the 
evaluation value for candidate area AcO, that is, the number of sample 
points Pel through Pe5 in candidate area AcO. In Embodiment 4, the 
sample points Pel through Pe5 on the sides of the candidate area are 
20 calculated as being in the candidate area. 
[0116] 

In Embodiment 4, the candidate evaluation value with the greatest 
number of evaluation values Hi is used as the image generation area. This 
embodiment allows the candidate area containing more areas with more 
25 overlapping frame image data to be selected as the image generation area. 
That is, in this embodiment, an area capable of generating high resolution 
still images can be used as the image generation area. 
[0117] 

E. Embodiment 5 

30 Figure 17 is a schematic illustration of the structure of an image 
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generator in Embodiment 5. Figure 18 is a flow chart of a procedure for 
generating still image data representing a still image from a plurality of 
frame images in motion picture data in Embodiment 5. In Embodiment 5, 
after the still image data has been generated in Step S8 of Figure 2, the user 
5 checks whether the generated image is good in Step S10. The functional 
component of the CPU 102 which carries out this function through the 
execution of the application program 95 is shown as the generated image 
confirmation component 102g in Figure 17. In the printing system in 
Embodiment 5, the points other than the process in Step S10 are the same 
10 as in Embodiment 1, including the hardware structure. 
[0118] 

Figure 19 is an illustration of a user interface display screen 
displayed on a display 110 in Step S10 in Figure 18. The frame image data 
obtained in Step S2 of Figure 18 (see Figure 3) and the still image data Ff 
15 generated in Step S8 are displayed side by side by the generated image 
confirmation component 102g. 
[0119] 

As illustrated in Figure 19, the still image data Ff generated in Step 
S8 is displayed as the same size as the frame image data F3 on the user 

20 interface. In Embodiment 5, candidate area Ac7 has been selected as the 
image generation area (see Figure 7). Candidate area Ac7 is an area 
smaller than the frame image data F3. Thus, when the still image data Ff 
generated with candidate area Ac7 is displayed on the display 110 on the 
same scale as the frame image data F3, the display will be smaller than the 

25 frame image data F3. In Figure 19, the area of the still image data Ff 
relative to the area of the frame image data F3 is indicated by the dashed 
line as area Ffo. 
[0120] 

The still image data Ff is enlarged more than the size that is 
30 displayed as the same scale as the frame image data F3 and is displayed as 
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the same size as the frame image data F3 on the user interface display 
screen in Figure 19. In Figure 19, the area when the still image data Ff is 
displayed on the display 110 as the same scale as the frame image data F3 
is represented by the dot-dash line as area Ffo2 on the still image data Ff. 
5 [0121] 

This is an example of when still image data Ff is generated with an 
area smaller than the frame image data F3. However, when the still image 
data Ff is generated with an area greater than the frame image data F3 
(such as candidate areas Ac9 to Acl2), the still image data Ff is shrunk 
10 smaller than when displayed as the same scale as the frame image data F3, 
and is displayed as the same size as the frame image data F3. 
[ 0122 ] 

The still image data Ff is displayed as the same scale as the frame 
image data F3 when the still image data Ff is generated with an area the 
15 same size as the frame image data F3 (such as candidate areas AcO to Ac4). 
The still image data Ff is thus displayed as the same size as the frame 
image data F3. 
[0123] 

This embodiment allows the user to easily compare the area of 
20 generated still image data Ff with the area of the image of the frame image 
data F3 selected by the user in Step S2. 
[0124] 

When the comparison reveals the displayed still image data Ff to be 
good, the user can use the mouse 130 to click the cursor Cs on the OK 

25 button on the screen as shown in the lower part of Figure 19. This 
concludes the process for generating still image data from the images in the 
frames of motion picture data. If, on the other hand, the comparison 
reveals the displayed still image data Ff to be unsuitable, the user can click 
the "Return" button shown at the bottom left side of Figure 19. This will 

30 restart the process for generating the still image data from Step S2 in 
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Figure 18. 
[0125] 

This embodiment allows the user to generate still image data having 
a desirable area after checking the contents of the still image data Ff that 
5 has been generated. 
[0126] 

F. Variants 

The invention is not limited to the preceding examples and 
embodiments, and can be worked in a variety of embodiments within the 
10 scope of the invention. The following variants are examples. 
[0127] 

(1) In Embodiment 1, the image from the still image data that has 
been generated is composed of a pixel density four times greater than that of 
the frame image data. However, the pixel density of the still image data is 

15 not limited to that level and may be another pixel density. That is, the 
pixel density forming the still image that is generated may be higher than 
that of the original image. Here, "higher pixel density" has the following 
meaning. That is, in cases where the first images and the second image are 
of the same subject, the second image will have a "higher pixel density" than 

20 the first images when the number of pixels used by the second image to 
represent the subject is more than the number of pixels used by the first 
images to represent the subject. 
[ 0128 ] 

(2) In Embodiment 1, one each of the candidate areas which had 
25 been shifted in mutually opposed directions based on candidate area AcO 

(which is an area equivalent to the area indicated by the user) were 
prepared. However, the number of these candidate areas is not limited to 1 
each and can be any number of 1 or more. It is preferable, however, to 
prepare the same number of candidate areas shifted in mutually opposed 
30 directions. 
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[0129] 

In Embodiment 1, one each of the candidate areas comprising areas 
expanded or shrunk based on the same reference point with respect to 
candidate area AcO were set as candidate areas. However, the number of 
5 these candidate areas is not limited to 1 each and can be any number of 1 or 
more. It is preferable, however, to prepare the same number of candidate 
areas comprising areas expanded or shrunk based on the same reference 
point. 
[0130] 

10 (3) In Embodiment 1, five sample points were set for each side of a 

candidate area. In Embodiment 4, five sample points were also set on the 
sides of the area of the image from the frame image data. However, the 
number of sample points is not limited to 5 and can be any number. The 
number preferably ranges from 5 to 21, however, and even more preferably 

15 from 9 to 17. The greater the number of sample points, the more detailed 
the evaluation of the candidate areas. However, the greater the number of 
sample points, the greater the calculations during the evaluation of the 
candidate areas. 
[0131] 

20 In Embodiment 2, the width of the evaluation area Aei was 1/20 of 

the long side of the rectangular candidate area. However, the width of the 
evaluation area Aei can be another value. The width of the portion of the 
evaluation area Aei near the short side of the candidate area is preferably 
predetermined to be no more than 1/5 the length L2 of the long side of the 

25 candidate area, and the width W2 of the portion of the evaluation area Ae 
near the long side of the candidate area is preferably predetermined to be no 
more than 1/5 of the length of the short side of the candidate area. The 
width Wl of the evaluation area Ae near the short side is even more 
preferably predetermined to be no more than 1/10 of L2. The width W2 of 

30 the evaluation area Ae near the long side is even more preferably 
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predetermined to be no more than 1/10 of the short side length LI of the 

candidate area.. 

[0132] 

In Embodiment 2, the image generation area was selected from the 
5 candidate areas based on the extent of overlap between the evaluation area 
AeO set at a predetermined width near the periphery inside the candidate 
area AcO and the area of the frame image data. However, the evaluation 
area may be an area set to a certain width near the outer periphery outside 
the candidate area. That is, the evaluation area can be an area near the 
10 profile of the candidate area when selecting the image generation area 
based on the extent of the overlap between the evaluation area and the area 
of the frame image data. 
[0133] 

"Near the profile of the candidate area" is defined as follows. The 
15 length of the longest line segment which can be included in the candidate 
area is referred to as a "first length." At that time, when a certain point is 
within 20% or less of the first length from the candidate area profile, that 
point is regarded as being "near the profile of the candidate area." 
[0134] 

20 In Embodiment 1, sample points were set on each side of candidate 

areas. However, a plurality of sample points may be set near the profile of 
the candidate areas, and the image generation area can be selected from the 
candidate areas based on the number of sample points within the area of the 
frame image data. 

25 [0135] 

The image generation area can also be selected based on the extent 
of the overlap between the candidate areas and the area of the image in the 
frames of image data. Such an embodiment allows the extent of the 
overlap to be assed in terms of the surface area of the overlapping sections. 
30 Based on the number of pixels in the frame image data which are included 
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in the aforementioned overlapping area, the extent of the overlap can be 

evaluated to select the image generation area. 

[0136] 

(4) In Embodiment 2, the evaluation value for the candidate areas 
5 was determined based on the number of pixels in the area included in the 
frame image data within evaluation area Aei. The number of pixels was 
counted based on the pixels in the frame image data. However, the number 
of pixels may also be counted based on the pixels in the image that is 
generated. The evaluation values of the candidate areas may thus be 
10 determined based on the number of pixels counted in this way. The 
evaluation values of the candidate areas may also be determined based on 
the surface area of the area included in the frame image data within the 
evaluation area Aei. 
[0137] 

15 (5) The target numerical value when selecting the image generation 

area from the candidate areas in Embodiments 2 to 4 was not input by the 
user. That is, the numerical value corresponding to the target evaluation 
value St in Embodiment 1 was not input by the user. However, the user 
may input such numerical values in Embodiments 2 through 4. In 

20 Embodiment 2, the user may input a Dt value, which is the target Di, 
through the keyboard 120 or mouse 130, and the candidate area with the 
evaluation value Di having the least difference from the Dt may be selected 
as the image generation area. Similarly, in Embodiment 3 or 4, the user 
may input the Gt value, which is the target Gi value, and the Ht value, 

25 which is the target Hi value, and the candidate area with the evaluation 
values Gi and Hi having the least difference from them may be selected as 
the image generation area. These embodiments allow the user to control 
the size and resolution of the image that is generated. 
[ 0138 ] 

30 For example, when a candidate area comprising a significant 
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reduction of candidate area AcO is set in Embodiment 2, the candidate area 
will be smaller than the area of the frame image data Fl through F5, and a 
greater proportion of parts of the evaluation area Aci is thus easier to 
include in the area of the frame image data Fl through F5. Such candidate 
5 areas therefore have a greater Di value, making it easier to select an image 
generation area. However, embodiments in which the user inputs the Dt, 
which is the target Di, and selects the candidate area having a Di with the 
least difference from the Dt as the image generation area can prevent 
candidate areas with low surface area from always being selected as the 
10 image generation area. The same is true of Embodiment 3. 
[ 0139 ] 

When a candidate area comprising a greatly expanded candidate 
area AcO is set in Embodiment 4, the candidate area will be greater than the 
area of the frame image data Fl through F5, making it easier for such 

15 candidate areas to include more sample points Pel through Pe5. Such 
candidate areas will therefore have a greater Hi value, making it easier to 
select the image generation area. However, embodiments in which the user 
inputs the Ht, which is the target Hi, and selects the candidate area having 
a Hi with the least difference from the Ht as the image generation area can 

20 prevent candidate areas with large surface area from always being selected 
as the image generation area. 
[0140] 

In Embodiment 1, there were five sample points set on each of the 
long and short sides of the candidate areas. The target evaluation value St 

25 was thus only one of 1 to 5. However, the number of sample points set on 
the sides of the candidate areas can be any number. When a different 
number of sample points are set on the short and long sides of the candidate 
areas, target evaluation values Stl and St2 can be set for the short and long 
sides, respectively, and the evaluation values of the candidate areas can be 

30 calculated based on the target evaluation values Stl and St2 and deviation 
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in the evaluation values Sij between the sides. 
[0141] 

(6) In Embodiment 1, the specified pixels were the pixels included in 
the same frame image data. However, the specified pixels are not limited 

5 to the area in the same frame image data. That is, the specified pixels can 
be any pixels near the target pixel. Here, "near the target pixel" refers to 
the range included in a circle having a radius twice as long as the width of 
the pixels in the frame image data based on the target pixel. The specified 
pixels are preferably the 3 or 4 pixels nearest the target pixel. 
10 [ 0142 ] 

(7) In the above examples, the shape of the area of the pixels in the 
still image data that is generated and the shape of the area of the pixels in 
the frame image data were similar. However, the area of the pixels in the 
still image data that is generated can be any shape. For example, the user 

15 can indicate or select the shape using the keyboard 120 or mouse 130. The 
candidate areas can thus be areas of the indicated shape which have been 
shifted vertically or laterally, or expanded or shrunk areas. 
[0143] 

(8) In the above examples, the pixels in the frame image data had 
20 red, green, and blue tone levels. However, the pixels of the frame image 

data can have tone levels of other combinations of colors, such as cyan, 

magenta, and yellow. 

[0144] 

(9) In Embodiment 5, the frame image data F3 obtained from the 
25 motion pictures and the still image data Ff that is generated were displayed 

on the display 110 (see Figure 19). However, the frame image data F3 
obtained from the motion pictures and the still image data Ff that is 
generated can be printed on a printer 22. Such an embodiment allows the 
user to compare the image area of the frame image data F3 and the image 
30 area of the still image data Ff that is generated. 
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[0145] 

That is, a printing system generating high resolution image data can 
output the low resolution image data, which is the starting material for 
generating high resolution image data, and high resolution image data 
5 generated from the low resolution image data by an output component 
capable of outputting image data in any form. The low resolution image 
data and the high resolution image data are preferably output in the same 
size. 
[ 0146 ] 

10 (10) In the above examples, the evaluation value for the candidate 

areas was determined based on the number of sample points or on the 
length of the portion included in the frame image data within the sides of 
the candidate areas Aci. However, the evaluation value for the candidate 
areas can be determined by other methods. The evaluation value for the 

15 candidate areas may be determined based on (i) the extent of the overlap 
between the candidate areas and the plurality of first images, and (ii) the 
target value representing the extent of the overlap between the image 
generation area and the plurality of first images. The evaluation values 
may be determined based on the deviation between the indicated value 

20 representing the extent of overlap between the candidate area and plurality 
of frame images (such as the evaluation value Sij on the sides of the 
candidate area in Embodiment l) and the target value (such as the target 
evaluation value St in Embodiment 1). 
[0147] 

25 (ll) In the above examples, part of the structure realized by 

hardware can be replaced by software (computer programs), and conversely 
part of the structure realized by software can be replaced by hardware. For 
example, the process involving the use of the frame data capturing 
component and the still image generator in Figure 1 can be done by a 

30 hardware circuit. 
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[ 0148 ] 

(12) Computer programs for running the above functions can be 
provided in the form of recordings on computer-readable recording media 
such as floppy disks and CD-ROMs. The host computer can read computer 

5 programs from the recording media and transfer them to an internal 
memory device or external memory device. Alternatively, computer 
programs may be provided to the host computer from a program provider 
through a communications circuit. When computer program functions are 
executed, computer programs stored in internal memory devices may be run 
10 by the microprocessor of the host computer. Computer programs recorded 
on recording media may also be directly run by the host computer. 
[ 0149 ] 

(13) In the present Specification, the concept of a host computer 
includes hardware devices and operating systems, and means hardware 

15 devices operated under the control of an operating system. Computer 
programs allow the functions of the aforementioned components to be run 
by such a host computer. Some of the aforementioned functions may be run 
by an operating system instead of application programs. 
[0150] 

20 (14) In the present invention, "computer-readable recording media" 

is not limited to portable recording media such as floppy disks and 
CD-ROMs, but also includes internal memory devices in computers, such as 
RAM or ROM, and external memory devices secured to computers, such as 
hard discs. 

25 [0151] 

(15) The Program product may be realized as many aspects. For 
example- 

(i) Computer readable medium, for example the flexible disks, the optical 
disk, or the semiconductor memories; 
30 (ii) Data signals, which comprise a computer program and are embodied 
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inside a carrier wave! 

(iii) Computer including the computer readable medium, for example the 
magnetic disks or the semiconductor memories; and 

(iv) Computer temporally storing the computer program in the memory 
5 through the data transferring means. 

[0152] 

(16) While the invention has been described with reference to preferred 
exemplary embodiments thereof, it is to be understood that the invention is 
not limited to the disclosed embodiments or constructions. On the contrary, 
10 the invention is intended to cover various modifications and equivalent 
arrangements. In addition, while the various elements of the disclosed 
invention are shown in various combinations and configurations, which are 
exemplary, other combinations and configurations, including more less or 
only a single element, are also within the spirit and scope of the invention. 
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